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.The Project on Instructional Criteria has attempted' 
to develop a aethodology for evaluation of computer service for 
instructional purposes. Through site visits,, letters, surveys of 
institutions and of xecent literature, and interviews, existing" 
practices were investigated. Alternative teohnigues were, designed and 
tested for measuring 'quality, quantity, and user Deeds. The outcoaes 
are supaari^ed and explained i^ a guidebook to the evaluation of 
coaputing, "Evaluating Instructional Computing: . Measuring Needs and 
Resources for Coiputing in Higher Education." It provides a review of 
i:he instructional uses of coaputing, recoaaends'' techniques for. the 
aeasurenent of the quality and quantity of coaputing, and suggests . 
techniques that evaluate alternatives on an individual caapus. 
(EMH) , 
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CRITfeRIA FOR ^ 
INSTRUCTIt)NAL COMPUTING EVALUATION- 
^ * IN HIGHER EDUCATrON 



The evdliKation of the cesoucces of hiqhec education in ' 
teems lof their impact on education has always, been d difficult 
matter. 'And so it is with cbmputjLnt| , one of the newest 
resources to be applied to the educational procegs. .There is 
general agreement that access to computing •^s' of value to 

students at all levels and in virtually all disciplines, that 

* pi- \ 

some kfnds of computing are of more value- than others, and that 

students wi'l 1 , i^ it is available, absorb considerable amounts- 

t 

of high qudi ity ^computing . Butr a^ttempts to quantify the need* ' 

• /• ^, j^. f ^ 

oc to define* kinds or (Quality of computing have usually 
cesulted in no consensus at all. Examination' of the practicii; 
of institutions provides 1 ittle guidance, since it covers ^uch* 
a-widfi rancfe as to reveal totally different approaches and 
totally different decisipqs. \ ' 

The subject is of more than casual or theoretical 
interest because it imping^es on many practical decisions at all 
levels in' the educational hierarchy. Administrators, 
legislators, and faculty are an^cious to identify, standards of 
quality and to define procedures for the evaluation of 
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educdtlondl computing: ideally in terms of acAdemic quality 
but -at least In teems oT the financial investment an 
institution* shcjuld be makinq to computers in Education • 

, Altftc>ugh periodic effoi^ts hdve been madt to survey 
colleges and univer sitles about expenditures c?id , hardware 
resourced, no serious effort to establish a nc'r^ative standard 
has taken place since the initial effort of the President's 
Science Advisory Bd^rcJ in 1967. No^effort at all has ^been. mad^ 
to establish stadidacds of quality independent of cost. 

The intent of the Project on Instructional Criteria has ^ 
been to dev^elop- a methodology for evaluatinq computer service 
for instructional purposes; these methods are intended to be . 
useful in curriculum, administrative, budgetary, and 
accreditation activities. * 



A. Objective s and pi an of attack r 
The .primary goal of 'the proj^ect was the definition of 

7 jr. 

useful guidelines for the assessment of computer service 
availability for students ibi highec- education . The effort can 
be seen as beir^ directed at finding answers to tl\e following 
three questions. 

What should be ' measured? In presenting facts about tneir 
use of' computers, ;Lnstitutiqns use many different measures^ It 
is important to consider what aspects of. the complex 
in/ter action between institutions, students', and computers 

si\ould be itiea'sured (and can be measured) in order to provide an 

' * ', . ' 

indication of conipute.c availability and its adequacy. 
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s .Altocndtiv,es include the amount of computing produced (in terms 

' ^ ^ . M 

of jobs oc hour s) , the cost of computing, the richness oc 
" . . . /' 

quality of computing produced. 

How should these be measure d? In order to measure^ any of 

these^ par ameters, some definitions are required. What 

constitutes a unit in student/compitt^r interaction? How is the 

size of a student job to be measured^ Can standardized # 

measures of cost be obtained in comparable form from a large 

\ 

number of academic computer facilities? 

How should the measur emen ts be eval uated ? If measures 
are defined along the lines sUg^gested by the first two 
questions/ the normative issue still has not been addressed: 
how much is- enough? No single norm is likely 'to be 
. satisfactory for all ' institutions. Requirements for comput|lri?q 
. avail dbil ity differ markedly from^ one discipline to another,. 



and between undergraduate and graducj^e instruction. Depending 
on commitment to different areas pf instruction and different 
institutional goals, commitmertt of resources to computing will 
also differ . ' ; \" ' ' 

The outcome of t1he study was planned to be a report,' with 
information presented in a for^that will be useful to 

0 

individuals in" education and government at several levels and 
for several purposes*: ,for self-evaluation, for pl,anning , for 
evaluating njfeeds and resources^. \ ^ 
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/ ^* PC oqcess, pc obi em s, and chdn gos 

The first stage of the investigation was an effort to 

\ discover what might already have been done elsewhere. It: 
seemed iijtpprtant to begin with a wide-ranging search fot 
methods, i^rocedures, and ideas that might serve as guide! ines\ 
Letters of inquiry' were addressed to leadinc^ institutions, site 
visits were made, and people were inter viewed*. In response to 
those inquiries and interviews , we received ^any words of 

4 • / ' ' . . 7' -" 

"encouragement but little concrete help. Most people are aware ' 
* ^ /« * *■ " " " 

-of the need for standards - but have little to offer by way of 

examples or methods. However., a few useful models did emer»^o. * 

In fxarallel with this investigation , an effort was made 

to approach the critical issues of what petople mc^asure and the 

, terms of • measurement they employ. Institution JvSkre aslped what 

kinds of units they use to account for computing re^urces and 

the catqgories they use to distinguish types of use. The ' 

results of this* survey reveal some Interesting aspects of 

institutional behavior, useful in developing guidel ines in " 

terms thab will be meaningful to many/institutions. The survey 

incidentally served to bring to the project copies of many 

internal documents and, reports that were helpful in the search 

for ex^ples of methods and procedures,. 



• ThB] third activity was a rev4ew of recent reports-and 
surveys that include quantitative data oh computer use in 
higher education. This provides a basis for aissessing what 

institution^ are currently doing and also for projecting the 

it ^ *^ *' 
changes taking pi ace in institutional commitments to computing. 



Foucthf al tec native techniques were desiqned and tested 
foe medsucing three important aspects of /4ducational computing: 
quality, quantity/, and uSer, needs. These techniques involve 
the ^bse of questionnair e§ and are described. in the report whifch 
is the major outcome of^the project. 

' A No major problems were encountered in these activities. 
Still y it must candidly be 'adm^itted that less infprmation, 
guidaTice, and help emerged, from the literature and people in 
the field than had originally been hoped.; Almost no 
institution can present a rigorous* and logical model fdt ^ ^ 

decision making. The fo^^mul as of those institutions usijiq 

* " » 'ft 

^onctete criteria for establishing levels of coniputinq support 

are ar bitr ar y.; and largely indefensible. They tend to be uneasy 

1- - -I- - ■ . " 

compromises between what faculty want and what institeuticfns can 
afford, with little r ef ec.enc^e to what students need. 

C. Significan t findings and outcome s. 

In the face of this situatibn, hopes that wi^could 
produce clear and absolute standards satisfactory to all we^e 
bound to be. defeated . *Concl usions as- "ungual if ied and 
uncompromising as those of the President's Science Advisory 
Board, which recommended spendinq $65 per un dergr adi^a te student 



per, year, are riot now defensible on the basis of the data and 
are not likely io be accepted by institutions or decision 
makers. More flexible guidelines, acknowledging the ^iffipdity 



\ 



of absolutes, will be more acceptable ' and more credible.\^) 

\ ■ • . ^ \ 



T'hoce fore, -the outcomes have be^n sumuidcizod at\^ 
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explained in the fg^cnv of a guidebook to the oval uatior^ -tof 
computing. Called Evaluat ing Instructio nal^ Comp uting ;■ . 
Measuring Needs and Resources for Computing in HigheA • " 
Education , it is addressed to those interested 4n' the| • . 
educational role of computing on the .camRus. It dndidat«s^how 
the adequacy of comptting on a par ticul ar * j;4mpus can b^ ° 
assessed and how an institution can go about finding, ar{*s wees to 
its owp, questions *^bout computing for studpnt and facgrljy use. . 
It begins by pp^^i^ding a review of the instructional usd'^ of 
computing^ it then recommend^ .techniques for the measurement 
of the quality ^d' quantity of computing; and' it suggests ' 
techniqueip that ev^l uate alternatives on an individual c^Lpus. 
More simfjly, , the questions addressed are these:. Whafisf, 
ins^tcuctional computing? Where ace we now? Where dd we ao 
^tr^-hee^-^^^ r^OiTt into thcefe parts^ more-or lees 



along the 1 ine^ of these three questions. ^ " 

What is ^ instr uct ion-ai cqnvputin^? These chapters treat 
the .nature and status of instructional computing in higher 
education. The need f5r„ student access to computing , thtl 
development of resources, and the evaluation of the . 
effectiveness of ^se are' the primary topics of the individual 
chapters. The ciiaptet summaries are as fojlows. 

"The,ne«d for standards"- presents an introduction to the 
questions abaut the evaluation of instructional computing lind » 
reviews the problem of achieving generally acceptable ^ 
standards. It concludes that, despite its newness, cOmputihq 
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hcis now tfecome an* intogcdj/ pac t of educdtion. Ear 1 y^estimates 

■ ^^-.-j^ . . . ' ' : • 

stcossod the iropoctdnce and impact computoc s 'would have on 
hiqhec education an4 cecommt^nded appcopciato levels of support. 
Mthough the importance and impact were not exaggerated, the 
recommended' levels for instr uctionjal ^computing have yet to be 
acFiieved . - , ^ \ 

/ "Surveys and estimates of U&e" summarizes the c3ata and 
conclusions of i^ajor surveys ofy.the u6e df computers in higher 
education. Two .very broad but important conclusiQns emerge. 
First, every year , more institutions use computers in 
education; more departments use com pouter s; an^ more students 
are exposed to the use of computer s. Second , oo a national 
basis,, expeaditurjes .for computers in higher education and tKe 
instructional use' of computers t?ontinue to rise.- 

"Instructional effectiveness" reviews the ways in which 
computers are applied ixv-Hrtm instructional process and the 
justification fo£,x£heir , use . It, concludes that "added cost. for 
added value",^fe^ the major justification. However, quantitative 
assessments of the effectiveness of computers in the 
educational process are, at best, sketchy and incomplete. 
Better information is biidly needed. / / 

Where are we now? The middle part of thf3 Report presents 
an assesisment of current use ar>d recommends techniques for the 
measurement of quantity and qofality of -computing resources 

provided for students to use . The three chapter s' that* make up 

* . • p ■ . 

this section are summarized below. 
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"What studtJnts do" considers the infocmatilbn avail able on 
the instructional techniques use^d in higher educ4tion and the^ 
way in which computers, courses, and students interact. The 
data indicate that most student computing is directed at 
learning about com{>uters and using computers to solve problems. 
Tutorial and other tecraiiques of computer-assisted instruction 
ace not widely us^d ih higher education, A further conclusion 
of the chapter is that, where computing is of high quality, 
where it is readil^y available, and where its use by students is 
entpuraged, nearljij^pll students (and faculty) will |nake som^ 
use of? It. The v^lue of student use is no^ necess'dc il y »in 
direct proportion ''to amount used or cost of the use. Most 
student users spend little time (or money) ireing.the computer 
but this use is a very important educational experience. At 
the end of this chapter, some hypotheses or rules of thumb ace 

suggested. Th^y'are intended to^simplify and assist the 

j ■ ' 

process of planning for student computer* use . - 

"Measuring quality" emphasizes the .fact that all 
computing is not the same and that some kinds of .Gbmputing 
resources ace too poor or of the wrong kind to be utilized 
effectively by students. Qualitative aspects of computing are, 
important if students are to use coip putting and if their use is 
to be productive in educational te^rms.* Qual\^t/ consider atiojia 
include: range of hardware and software available, ^ ^ 
convenience, accessibility, availability of documentation and^' 
assistance, ^nd simplicity of access. General c<jmpus awareness 
of computlrtg may also be a good measure of quality and 
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C matucity, / This 'chapter suggests a 'technique foe measuring and 
assessing quality. . . ^ 



"Measuring quantity" considers the inadequacy of much of 

the availablcT information and the lack of generally accepted / 

standards of measurement in this field. Three specific 

■ ■ • .... I „ 

conclusions are put fdrWard in the chapter • First, if v?e are 

to understand iastr uc tional computing better , we need better 

measures and better records of student use • Second , in these 

records,, it is .important * that we be able to distinguish (at 

least) two-levels\, undergraduate and graduate; and (at leasts 

distinguish major Wasses of users, such as computer science, 

V sqience and engineer ing , and other students, . Th^rd , computing- 

must be measured sc^m^e terms other than (or in addition to) 

dollars, 

Wher^e do we from here? The last chapters speak 
directly to the techniques of decision making on the individual 
campus, and recommend methods for rationalizing and formalizing 
these . techniques. ^ 

- "Peer groups and institutional estimates" presents .a 
technique for gathering information on the use of computers at 
"comparable" institutions, since many institutions find this ' an 
iifiportant wky of establ ishirtg a norm. 

/ "Self-sVudy: assessing the need" reviews techniques of 
evaludTt'ing the consensus of those whose opinions matter 'to the 
institutioni the facul^^y or- some other group. A specific 
instrument'^and *its method of appl icatl^n ar e presented. 
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"Priorities, al location,/ anc3 decisionsy/tdV^'X^nofehor . 
brodd iook/dt the* c:dmpus . situdtion with regard to \;^orhputing , 
sketches /sever al / alter ndtive/s, dnd^ makes r ecommenddt^ions for <^ 
future "study • - T. v " 

^ Deferent institutions have differ 6nt ways to go and it / 
is impossit^le to set down rule^ universally applicable to all 
univer siti4s| and cdl leges. But the 
so diverse J. All of them need to be 
computer-aided world. Institutions need to .formulate goals and 
establish resources in terms /of this need . What difference 
does computing make in educa/tion?\> As is the case with most 
educational activities, we can measure^ thf^ inputs but /the 
outputs, the results, the things that matter ar.e more elusive. 
More efforts to measure the impact of computers on education 
•are irnmrtant. But taking advantage- of ihi2 impact and 




need s o'f stud en t s ar e no t^ 
educated in h ■ " 



improving the use of computers in instruction ace more 



impor t^f^nt .stiJ I . The study urges better measures for both the 
computing itself and its influence on education . But it also 
advocates the ira^pr fcance o-f more and better 'tomputinq thant mciny 
students have accesfe to today. ' " ■* . 

*PL>t>3 ica tion 



Several oral repox.ts on the * proa r ess of tbe tesearch have 
beien preser^ted, as followsr j , ' 1 , ^' 

;r "Compute^r Access fdr Stud[ents: ^ jjuantlti* and Ouaiity," 

presented at meeting of Association for the DfiVelopment of 
Computer-based Instr uc tiona] Systems, .Santa Barbara, 
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Cdlifocnld, 27 JdriudCy 1976. 

/ " , ^1 ♦ . 

» .» ' * 

^ ^ "Pricing dnd Allocation Schemes," pcesent«id dt meeting cjf 
ACM Special Interest Group on Computer Uses in Educdtion, 
Anaheim, Californid, 11 Pebrudcy 1$76. 

" Estdbl ishing Cr iter id for Computers in Education," ACM 
* > ' < • '• 

Special Interest Group on University Computer . Center s, St. 

Lauis, Missouri, 9 April 1976./ 

T^lks on the conclusions of the researclY'Var e also 
scheduled for delivery at the National ACM meeting m. Houston 
in October; and a meeting on User Sec vie e?s\ sponsor ecf* by the ACM 

Specidl Interest Grpup on University Compater Centers in Tucson 

• \ * * ^ ' * 

in NovemberL > 

\ ■ ' ■ ^ . " f ^ 

^ All o§ the interim r^eports of the project have been 

superceded by a major publ ication , Eval uating Instriic tional 

C omputi ng , which repor'ts on^the research ^nd its findings .at 

some length. This report will be published by t\\Q Univer^it;y 

of California,, Irvine, and is now in the press. It will be 

available in August ^of 1976 . ' - * . - ' ^ 
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